Robotic system with piece-loss management mechanism

ABSTRACT

A method for operating a robotic system that includes calculating a base motion plan, wherein the base motion plan includes a sequence of commands or settings, or a combination thereof, that operates a robotic arm and a gripper to transfer a target object from a start location to a task location; receiving a contact measure while executing the base motion plan, wherein the contact measure represents an amount of grip of the gripper on the target object; and generating one or more actuator commands/settings that deviate from the base motion plan when the contact measure fails to satisfy a threshold, wherein the one or more actuator commands/settings thereof are configured to operate the robotic arm, the gripper, or a combination thereof to execute one or more response actions not included in the base motion plan.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/414,396 filed May 16, 2019, issued as U.S. Pat. No. 10,532,462, whichis a continuation of U.S. patent application Ser. No. 16/252,383 filedJan. 18, 2019, issued as U.S. Pat. No. 10,335,947, both of which areincorporated herein by reference in their entirety.

TECHNICAL FIELD

The present technology is directed generally to robotic systems and,more specifically, to systems, processes, and techniques for detectingand managing piece-loss scenarios.

BACKGROUND

With their ever-increasing performance and lowering cost, many robots(e.g., machines configured to automatically/autonomously executephysical actions) are now extensively used in many fields. Robots, forexample, can be used to execute various tasks (e.g., manipulate ortransfer an object through space) in manufacturing and/or assembly,packing and/or packaging, transport and/or shipping, etc. In executingthe tasks, the robots can replicate human actions, thereby replacing orreducing human involvements that are otherwise required to performdangerous or repetitive tasks.

However, despite the technological advancements, robots often lack thesophistication necessary to duplicate human sensitivity and/oradaptability required for executing more complex tasks. For example,robot end-effectors (e.g., robotic hands or grippers) often havedifficulty grabbing objects with relatively soft and/or irregularsurfaces due to lack of sensitivity in contact sensors and/orinsufficient granularity in force control. Also, for example, robotsoften cannot account for conditions or situations outside of thetargeted conditions/scenario due to lack of adaptability. Accordingly,there remains a need for improved techniques and systems for controllingand managing various aspects of the robots.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of an example environment in which a roboticsystem with a piece-loss management mechanism may operate.

FIG. 2 is a block diagram illustrating the robotic system in accordancewith one or more embodiments of the present technology.

FIG. 3A is an illustration of an example of a grip state in accordancewith one or more embodiments of the present technology.

FIG. 3B is an illustration of a further example of a grip state inaccordance with one or more embodiments of the present technology.

FIG. 4 is a top view illustrating an example task executed by therobotic system in accordance with one or more embodiments of the presenttechnology.

FIG. 5 is a flow diagram for operating the robotic system of FIG. 1 inaccordance with one or more embodiments of the present technology.

DETAILED DESCRIPTION

Systems and methods for a robotic system with a piece-loss managementmechanism are described herein. The robotic system (e.g., an integratedsystem of devices that execute one or more designated tasks) configuredin accordance with some embodiments provides piece-loss management byimplementing granular control/manipulation of a target object accordingto a contact measure. Using one or more sensors, the robotic system candetermine the contact measure that represents a quantized amount ofcontact corresponding to a stability of the target object relative to anend-effector. In other words, the contact measure can represent aquantized amount of grip that the end-effector has on the target object.Based on the contact measure, the robotic system can regrip the targetobject, execute a controlled drop of the target object at a designatedlocation, select and/or adjust a motion plan, or a combination thereof.

The robotic system can be configured to execute a task based onmanipulating (e.g., physically displacing and/or reorienting) the targetobject. For example, the robotic system can sort or relocate variousobjects based on picking the target object from a source location (e.g.,a bin, a pallet, or a conveyer belt) and moving it to a destinationlocation. In some embodiments, for manipulating the target object, therobotic system can include a gripper operably connected to a robot arm.The gripper can be configured to affix the target object relative to therobot arm. In other words, the robotic system can operate the gripper(via, e.g., one or more associated motors/actuators and sensors) to grabthe target object and hold it relative to the robot arm. The roboticsystem can similarly operate the robot arm to manipulate the gripper,the target object held by the gripper, or a combination thereof.

To execute the task, in some embodiments, the robotic system can includean imaging device (e.g., a camera, an infrared sensor/camera, a radar, alidar, etc.) used to identify a location and/or a pose (e.g., a restingorientation) of the target object and/or the environment around thetarget object. According to the location, the pose, or a combinationthereof, the robotic system can implement a motion plan (e.g., asequence of controls for the actuators for moving one or more linksand/or joints) to execute the task. For example, for sorting and/orrelocating the target object, the motion plan can correspond to grippingthe target object initially at the source location, manipulating itacross space, and placing it at the destination location.

In some situations, however, the grip (e.g., a degree of attachment) ofthe gripper on the target object can fail during execution of the task.As a result, the target object may be displaced or shifted relative tothe gripper. In some cases, grip failure can lead to a lost piece (e.g.,the target object that was not placed at the destination location and/orin an intended pose), such as when the gripper drops or loses control ofthe target object during the manipulation. Failed grip can be caused by,for example, forces applied to the target object and/or inertia of thetarget object, shifting of the target object (e.g., a box or contentinside the box), or a combination thereof resulting from themanipulation. Also, for example, failed grip can be caused by acalibration error in the imaging mechanism.

Traditional manipulators (e.g., picker robots) often implement arelatively fixed motion plan that does not deviate from the task. Whiletraditional manipulators may account for different locations and/orposes of an object, once the object is picked up, the motion plan tomanipulate the object to a destination location/orientation remainsfixed. In contrast, various embodiments of the robotic system describedbelow are configured to determine (e.g., when the target object isgripped and/or while executing the task) a contact measure (e.g., anamount or a degree of the grip) and implement granularcontrol/manipulation of the target object accordingly. Determination ofthe contact measure and the granular control/manipulation are describedin detail below.

In the following, numerous specific details are set forth to provide athorough understanding of the presently disclosed technology. In otherembodiments, the techniques introduced here can be practiced withoutthese specific details. In other instances, well-known features, such asspecific functions or routines, are not described in detail in order toavoid unnecessarily obscuring the present disclosure. References in thisdescription to “an embodiment,” “one embodiment,” or the like mean thata particular feature, structure, material, or characteristic beingdescribed is included in at least one embodiment of the presentdisclosure. Thus, the appearances of such phrases in this specificationdo not necessarily all refer to the same embodiment. On the other hand,such references are not necessarily mutually exclusive either.Furthermore, the particular features, structures, materials, orcharacteristics can be combined in any suitable manner in one or moreembodiments. It is to be understood that the various embodiments shownin the figures are merely illustrative representations and are notnecessarily drawn to scale.

Several details describing structures or processes that are well-knownand often associated with robotic systems and subsystems, but that canunnecessarily obscure some significant aspects of the disclosedtechniques, are not set forth in the following description for purposesof clarity. Moreover, although the following disclosure sets forthseveral embodiments of different aspects of the present technology,several other embodiments can have different configurations or differentcomponents than those described in this section. Accordingly, thedisclosed techniques can have other embodiments with additional elementsor without several of the elements described below.

Many embodiments or aspects of the present disclosure described belowcan take the form of computer- or controller-executable instructions,including routines executed by a programmable computer or controller.Those skilled in the relevant art will appreciate that the disclosedtechniques can be practiced on computer or controller systems other thanthose shown and described below. The techniques described herein can beembodied in a special-purpose computer or data processor that isspecifically programmed, configured, or constructed to execute one ormore of the computer-executable instructions described below.Accordingly, the terms “computer” and “controller” as generally usedherein refer to any data processor and can include Internet appliancesand handheld devices (including palm-top computers, wearable computers,cellular or mobile phones, multi-processor systems, processor-based orprogrammable consumer electronics, network computers, mini computers,and the like). Information handled by these computers and controllerscan be presented at any suitable display medium, including a liquidcrystal display (LCD). Instructions for executing computer- orcontroller-executable tasks can be stored in or on any suitablecomputer-readable medium, including hardware, firmware, or a combinationof hardware and firmware. Instructions can be contained in any suitablememory device, including, for example, a flash drive, USB device, and/orother suitable medium.

The terms “coupled” and “connected,” along with their derivatives, canbe used herein to describe structural relationships between components.It should be understood that these terms are not intended as synonymsfor each other. Rather, in particular embodiments, “connected” can beused to indicate that two or more elements are in direct contact witheach other. Unless otherwise made apparent in the context, the term“coupled” can be used to indicate that two or more elements are ineither direct or indirect (with other intervening elements between them)contact with each other, or that the two or more elements co-operate orinteract with each other (e.g., as in a cause-and-effect relationship,such as for signal transmission/reception or for function calls), orboth.

Suitable Environments

FIG. 1 is an illustration of an example environment in which a roboticsystem 100 with a piece-loss management mechanism may operate. Therobotic system 100 includes one or more structures (e.g., robots)configured to execute one or more tasks. Aspects of the piece-lossmanagement mechanism can be practiced or implemented by the variousstructures.

For the example illustrated in FIG. 1, the robotic system 100 caninclude an unloading unit 102, a transfer unit 104, a transport unit106, a loading unit 108, or a combination thereof in a warehouse or adistribution/shipping hub. Each of the units in the robotic system 100can be configured to execute one or more tasks. The tasks can becombined in sequence to perform an operation that achieves a goal, suchas to unload objects from a truck or a van for storage in a warehouse orto unload objects from storage locations and load them onto a truck or avan for shipping. For another example, the task can include movingobjects from one container to another container. Each of the units canbe configured to execute a sequence of actions (e.g., operating one ormore components therein) to execute a task.

In some embodiments, the task can include manipulation (e.g., movingand/or reorienting) of a target object 112 (e.g., boxes, cases, cages,pallets, etc.) from a start location 114 to a task location 116. Forexample, the unloading unit 102 (e.g., a devanning robot) can beconfigured to transfer the target object 112 from a location in acarrier (e.g., a truck) to a location on a conveyor belt. Also, thetransfer unit 104 (e.g., a palletizing robot) can be configured totransfer the target object 112 from a location on the conveyor belt to alocation on the transport unit 106, such as for loading the targetobject 112 on a pallet on the transport unit 106. For another example,the transfer unit 104 (e.g., a piece-picking robot) can be configured totransfer the target object 112 from one container to another container.In completing the operation, the transport unit 106 can transfer thetarget object 112 from an area associated with the transfer unit 104 toan area associated with the loading unit 108, and the loading unit 108can transfer the target object 112 (by, e.g., moving the pallet carryingthe target object 112) from the transfer unit 104 to a storage location(e.g., a location on the shelves). Details regarding the task and theassociated actions are described below.

For illustrative purposes, the robotic system 100 is described in thecontext of a shipping center; however, it is understood that the roboticsystem 100 can be configured to execute tasks in otherenvironments/purposes, such as for manufacturing, assembly, packaging,healthcare, and/or other types of automation. It is also understood thatthe robotic system 100 can include other units, such as manipulators,service robots, modular robots, etc., not shown in FIG. 1. For example,in some embodiments, the robotic system 100 can include a depalletizingunit for transferring the objects from cage carts or pallets ontoconveyors or other pallets, a container-switching unit for transferringthe objects from one container to another, a packaging unit for wrappingthe objects, a sorting unit for grouping objects according to one ormore characteristics thereof, a piece-picking unit for manipulating(e.g., for sorting, grouping, and/or transferring) the objectsdifferently according to one or more characteristics thereof, or acombination thereof.

Suitable System

FIG. 2 is a block diagram illustrating the robotic system 100 inaccordance with one or more embodiments of the present technology. Insome embodiments, for example, the robotic system 100 (e.g., at one ormore of the units and/or robots described above) can includeelectronic/electrical devices, such as one or more processors 202, oneor more storage devices 204, one or more communication devices 206, oneor more input-output devices 208, one or more actuation devices 212, oneor more transport motors 214, one or more sensors 216, or a combinationthereof. The various devices can be coupled to each other via wireconnections and/or wireless connections. For example, the robotic system100 can include a bus, such as a system bus, a Peripheral ComponentInterconnect (PCI) bus or PCI-Express bus, a HyperTransport or industrystandard architecture (ISA) bus, a small computer system interface(SCSI) bus, a universal serial bus (USB), an IIC (I2C) bus, or anInstitute of Electrical and Electronics Engineers (IEEE) standard 1394bus (also referred to as “Firewire”). Also, for example, the roboticsystem 100 can include bridges, adapters, controllers, or othersignal-related devices for providing the wire connections between thedevices. The wireless connections can be based on, for example, cellularcommunication protocols (e.g., 3G, 4G, LTE, 5G, etc.), wireless localarea network (LAN) protocols (e.g., wireless fidelity (WIFI)),peer-to-peer or device-to-device communication protocols (e.g.,Bluetooth, Near-Field communication (NFC), etc.), Internet of Things(IoT) protocols (e.g., NB-IoT, LTE-M, etc.), and/or other wirelesscommunication protocols.

The processors 202 can include data processors (e.g., central processingunits (CPUs), special-purpose computers, and/or onboard servers)configured to execute instructions (e.g. software instructions) storedon the storage devices 204 (e.g., computer memory). The processors 202can implement the program instructions to control/interface with otherdevices, thereby causing the robotic system 100 to execute actions,tasks, and/or operations.

The storage devices 204 can include non-transitory computer-readablemediums having stored thereon program instructions (e.g., software).Some examples of the storage devices 204 can include volatile memory(e.g., cache and/or random-access memory (RAM) and/or non-volatilememory (e.g., flash memory and/or magnetic disk drives). Other examplesof the storage devices 204 can include portable memory drives and/orcloud storage devices.

In some embodiments, the storage devices 204 can be used to furtherstore and provide access to processing results and/or predetermineddata/thresholds. For example, the storage devices 204 can store masterdata that includes descriptions of objects (e.g., boxes, cases, and/orproducts) that may be manipulated by the robotic system 100. In one ormore embodiments, the master data can include a dimension, a shape(e.g., templates for potential poses and/or computer-generated modelsfor recognizing the object in different poses), a color scheme, animage, identification information (e.g., bar codes, quick response (QR)codes, logos, etc., and/or expected locations thereof), an expectedweight, or a combination thereof for the objects expected to bemanipulated by the robotic system 100. In some embodiments, the masterdata can include manipulation-related information regarding the objects,such as a center-of-mass location on each of the objects, expectedsensor measurements (e.g., for force, torque, pressure, and/or contactmeasurements) corresponding to one or more actions/maneuvers, or acombination thereof. Also, for example, the storage devices 204 canstore object tracking data. In some embodiments, the object trackingdata can include a log of scanned or manipulated objects. In someembodiments, the object tracking data can include imaging data (e.g., apicture, point cloud, live video feed, etc.) of the objects at one ormore locations (e.g., designated pickup or drop locations and/orconveyor belts). In some embodiments, the object tracking data caninclude locations and/or orientations of the objects at the one or morelocations.

The communication devices 206 can include circuits configured tocommunicate with external or remote devices via a network. For example,the communication devices 206 can include receivers, transmitters,modulators/demodulators (modems), signal detectors, signalencoders/decoders, connector ports, network cards, etc. Thecommunication devices 206 can be configured to send, receive, and/orprocess electrical signals according to one or more communicationprotocols (e.g., the Internet Protocol (IP), wireless communicationprotocols, etc.). In some embodiments, the robotic system 100 can usethe communication devices 206 to exchange information between units ofthe robotic system 100 and/or exchange information (e.g., for reporting,data gathering, analyzing, and/or troubleshooting purposes) with systemsor devices external to the robotic system 100.

The input-output devices 208 can include user interface devicesconfigured to communicate information to and/or receive information fromhuman operators. For example, the input-output devices 208 can include adisplay 210 and/or other output devices (e.g., a speaker, a hapticscircuit, or a tactile feedback device, etc.) for communicatinginformation to the human operator. Also, the input-output devices 208can include control or receiving devices, such as a keyboard, a mouse, atouchscreen, a microphone, a user interface (UI) sensor (e.g., a camerafor receiving motion commands), a wearable input device, etc. In someembodiments, the robotic system 100 can use the input-output devices 208to interact with the human operators in executing an action, a task, anoperation, or a combination thereof.

The robotic system 100 can include physical or structural members (e.g.,robotic manipulator arms) that are connected at joints for motion (e.g.,rotational and/or translational displacements). The structural membersand the joints can form a kinetic chain configured to manipulate anend-effector (e.g., the gripper) configured to execute one or more tasks(e.g., gripping, spinning, welding, etc.) depending on the use/operationof the robotic system 100. The robotic system 100 can include theactuation devices 212 (e.g., motors, actuators, wires, artificialmuscles, electroactive polymers, etc.) configured to drive or manipulate(e.g., displace and/or reorient) the structural members about or at acorresponding joint. In some embodiments, the robotic system 100 caninclude the transport motors 214 configured to transport thecorresponding units/chassis from place to place.

The robotic system 100 can include the sensors 216 configured to obtaininformation used to implement the tasks, such as for manipulating thestructural members and/or for transporting the robotic units. Thesensors 216 can include devices configured to detect or measure one ormore physical properties of the robotic system 100 (e.g., a state, acondition, and/or a location of one or more structural members/jointsthereof) and/or for a surrounding environment. Some examples of thesensors 216 can include accelerometers, gyroscopes, force sensors,strain gauges, tactile sensors, torque sensors, position encoders, etc.

In some embodiments, for example, the sensors 216 can include one ormore imaging devices 222 (e.g., 2-dimensional and/or 3-dimensionalcameras including visual and/or infrared cameras, lidars, radars, and/orother distance-measuring or imaging devices) configured to detect thesurrounding environment. The imaging device 214 can generate arepresentation of the detected environment, such as a digital imageand/or a point cloud, used for implementing machine/computer vision(e.g., for automatic inspection, robot guidance, or other roboticapplications). As described in further detail below, the robotic system100 (via, e.g., the processors 202) can process the digital image and/orthe point cloud to identify the target object 112 of FIG. 1, the startlocation 114 of FIG. 1, the task location 116 of FIG. 1, a pose of thetarget object 112 of FIG. 1, or a combination thereof. For manipulatingthe target object 112, the robotic system 100 (e.g., via the variousunits) can capture and analyze an image of a designated area (e.g.,inside the truck, inside the container, or a pickup location for objectson the conveyor belt) to identify the target object 112 and the startlocation 114 thereof. Similarly, the robotic system 100 can capture andanalyze an image of another designated area (e.g., a drop location forplacing objects on the conveyor belt, a location for placing objectsinside the container, or a location on the pallet for stacking purposes)to identify the task location 116.

Also, for example, the sensors 216 can include position sensors 224(e.g., position encoders, potentiometers, etc.) configured to detectpositions of structural members (e.g., the robotic arms and/or theend-effectors) and/or corresponding joints of the robotic system 100.The robotic system 100 can use the position sensors 224 to tracklocations and/or orientations of the structural members and/or thejoints during execution of the task.

In some embodiments, the sensors 216 can include contact sensors 226(e.g., pressure sensors, force sensors, strain gauges,piezoresistive/piezoelectric sensors, capacitive sensors,elastoresistive sensors, and/or other tactile sensors) configured tomeasure a characteristic associated with a direct contact betweenmultiple physical structures or surfaces. The contact sensors 226 canmeasure the characteristic that corresponds to a grip of theend-effector (e.g., the gripper) on the target object 112. Accordingly,the contact sensors 226 can output a contact measure that represents aquantified measure (e.g., a measured force, torque, position, etc.)corresponding to a degree of contact or attachment between the gripperand the target object 112. For example, the contact measure can includeone or more force or torque readings associated with forces applied tothe target object 112 by the end-effector. Details regarding the contactmeasure are described below.

As described in further detail below, the robotic system 100 (via, e.g.,the processors 202) can implement different actions to accomplish thetask based on the contact measure. For example, the robotic system 100can regrip the target object 112 if the initial contact measure is belowa threshold. Also, the robotic system 100 can intentionally drop thetarget object 112, adjust the task location 116, adjust a speed or anacceleration for the action, or a combination thereof if the contactmeasure falls below a threshold during execution of the task.

Contact Measurements

FIG. 3A and FIG. 3B illustrate examples of grip states in accordancewith one or more embodiments of the present technology. In someembodiments, the robotic system 100 of FIG. 1 (e.g., at one or moreunits, such as the palletizing/depalletizing robot, the picker robot,etc. described above) can include an end-effector (e.g., a gripper)connected to a robotic arm 304. The robotic arm 304 can includestructural members and/or joints between the members configured tomanipulate the end-effector. The end-effector can be manipulated byoperating the actuation devices 212 of FIG. 2 connected to thestructural members and/or the joints of the robotic arm 304.

In some embodiments, the end-effector (e.g., the gripper) can beconfigured to grip an object, thereby securing it or affixing itrelative to the end-effector. The end-effector can also be operated(e.g., for grabbing and/or releasing) by operating one or more of theactuation devices 212 associated with or attached to one or moreportions of the end-effector.

In one or more embodiments, as illustrated in FIG. 3A, the end-effectorcan include a gripper 302 (e.g., an astrictive or a suction gripper)configured to hold or affix the target object 112 via attractive forces,such as achieved by forming and maintaining a vacuum condition betweenthe gripper 302 and the target object 112. For example, the gripper 302can include a set of suction cups 306 configured to contact a surface ofthe target object 112 and form/retain the vacuum condition in the spacesbetween the suction cups 306 and the surface. The vacuum condition canbe created when the gripper 302 is lowered via the robotic arm 304,thereby pressing the suction cups 306 against the surface of the targetobject 112 and pushing out gases between the opposing surfaces. When therobotic arm 304 lifts the gripper 302, a difference in pressure betweenthe spaces inside the suction cups 306 and the surrounding environmentcan keep the target object 112 attached to the suction cups 306.Accordingly, a degree of grip or attachment of the gripper 302 on thetarget object 112 can be based on the number of the suction cups 306successfully creating and holding the vacuum condition.

Various factors may prevent the suction cups 306 from successfullycreating and holding the vacuum condition. For example, a calibrationerror in the imaging devices 222 of FIG. 2 can cause the gripper 302 tobe misplaced or misaligned relative to the target object 112. As such,one or more of the suction cups 306 may not properly contact (e.g., asillustrated by a separation gap 322) the surface of the target object112 to create and hold the vacuum condition. Also, unexpecteddeformities or particulates on the surface of the target object 112 mayprevent one or more of the suction cups 306 from forming a sealed spaceon the surface of the target object 112 that holds the vacuum condition.Also, during manipulation of the target object 112, one or more of thesuction cups 306 may experience forces resulting from movement inertiaand/or shifting (e.g., a box or content inside the box) of the targetobject 112. When the experienced forces are greater than the integrityof the formed seal, the suction cups 306 may fail to hold the vacuumcondition.

In some embodiments, the gripper 302 includes the contact sensors 226 ofFIG. 2 (e.g., one or more force, pressure, torque, and/or other tactilesensors) configured to determine a contact measure 312. The contactsensors 226 can generate the contact measure 312 as a representation ofa degree of attachment of the gripper 302 to the target object 112. Inother words, the contact measure 312 can represent a measure or anamount of grip of the end-effector on the target object 112. Forexample, the contact sensors 226 can include touch or tactile sensorsconfigured to indicate whether sensed surfaces are contacting anothersurface and/or configured to determine the size of the surface areacontacting another surface. Also, the contact sensors 226 can includepressure sensors configured to measure the pressure (e.g., the vacuumcondition) inside the suction cups 306. Also, the contact sensors 226can include linear force sensors configured to measure the weight (e.g.,as illustrated by dashed linear arrows) of the target object 112 borneor supported by the suction cups 306. Further, the contact sensors 226can include torque sensors configured to measure torque (e.g., asillustrated by dashed curved arrows) on the suction cups 306, thegripper 302, and/or the robotic arm 304. In comparison to a fullygripped state, the torque measurements can change (e.g., increase) whensome of the suction cups 306 (e.g., the peripherally located ones) failto hold the vacuum condition. According to the type and/or location ofthe contact sensors 226, the contact measure 312 can correspond to a sumor an average of the measurements (e.g., the internal pressure, thelinear force, and/or the torque) across the suction cups 306, a quantityof the suction cups 306 and/or locations thereof with measurementssatisfying a vacuum threshold, or a combination thereof.

As an illustrative example, FIG. 3A shows suction cups on a distal end(i.e., located on the right side of FIG. 3A) of the gripper 302 havinggrip on the target object 112 (as illustrated by arrows traversingthrough to the target object 112). In contrast, suction cups on aproximal end (i.e., located on the left side of FIG. 3A) of the gripper302 are shown as being separated by the separation gap 322. Accordingly,linear force sensors corresponding to the suction cups on the distal endcan determine non-zero readings associated with the weight borne by thedistal suction cups. Also, linear force sensors corresponding to thesuction cups on the proximal end can determine zero or near-zeroreadings due to the failed grip. Further, due to the uneven distributionof the force, a torque sensor associated with the gripper 302 candetermine a non-zero reading.

In comparison, if all of the suction cups 306 established and maintainedthe vacuum condition with the surface of the target object 112, thelinear force readings would have a non-zero magnitude at all of thesuction cups 306 and/or deviations between of the linear force readingswould be within a relatively small range. Further, since the weightwould be distributed in a substantially even manner across the suctioncups 306, the torque measured at the gripper 302 would be closer to azero value.

As such, the robotic system 100 can use the above examples of thecontact measure 312 as a representation of grip of the gripper 302 onthe target object 112. For example, the deviations in the linear forcereadings and/or torque readings can inversely represent the gripstrength. In other words, greater deviations from expected readings(e.g., near-zero deviations in the linear force measurements and/ornear-zero torque measurements can correspond to a strong grip) cancorrespond to weaker grip. In some embodiments, the robotic system 100can further use a lookup/translation table, an equation, a process, or acombination thereof for translating/transposing the expected readingsaccording to different orientations (e.g., poses) of the gripper 302 andthe target object 112. In some embodiments, the master data can includethe expected readings for each of the different orientations of thegripper 302 and the target object 112. The robotic system 100 can usethe expected readings to evaluate or process the contact measure 312according to the orientation of the gripper 302 and the target object112.

In some embodiments, as illustrated in FIG. 3B, the robotic system 100of FIG. 1 (e.g., at one or more units, such as thepalletizing/depalletizing robot, the picker robot, etc. described above)can include a gripper 352 (e.g., an impactive gripper) configured tophysically grasp the target object 112 via direct impact. For example,the gripper 352 can include gripper jaws 356 configured to grip thetarget object 112 based on applying opposing or compressing forces onthe target object 112. The target object 112 can be gripped based on theresulting friction between contacting surfaces of the gripper jaws 356and the target object 112.

Various factors may prevent the gripper jaws 356 from successfullygripping the target object 112. For example, a calibration error in theimaging devices 222 of FIG. 2 can cause the gripper 352 to be misplacedor misaligned relative to the target object 112. As such, the gripperjaws 356 may contact unintended portions of the target object 112, suchas where the surface characteristics reduce the resulting friction.Also, unexpected deformities or particulates on the surface of thetarget object 112 may reduce the resulting friction. Also, duringmanipulation of the target object 112, the gripper jaws 356 mayexperience forces resulting from movement inertia and/or shifting (e.g.,a box or content inside the box) of the target object 112. When theexperienced forces are greater than the friction force, the gripper jaws356 may fail to hold the vacuum condition.

In some embodiments, the gripper 352 includes the contact sensors 226 ofFIG. 2 (e.g., one or more force, pressure, and/or torque sensors)configured to determine the contact measure 312. For example, thecontact sensors 226 can include tactile sensors that indicate directcontact between sensed surfaces and other surfaces and/or measure thesize of the contact area. Also, the contact sensors 226 can includepressure or tactile sensors on a contacting surface of the gripper jaws356 configured to measure a force exerted on the target object 112 bythe gripper jaws 356. Further, the contact sensors 226 can includelinear force sensors configured to measure the weight of the targetobject 112. When the grip fails, the target object 112 can slip, whichcan result in a reduction in the weight sensed by the linear forcesensors.

As illustrated in FIG. 3B, for example, improper or failed grip canresult in the target object 112 remaining stationary (i.e., sliding downrelative to the gripper jaws 356) as the gripper 352 moves upward in anattempt to lift the target object 112. Accordingly, the weight or theforce measured by the linear force sensor can be less than the actualweight or a portion thereof (e.g., about half on each jaw) that wouldhave been measured if the grip were sufficient and the target object 112had remained fixed relative to the gripper jaws 356. As an illustrativeexample, the robotic system 100 can track measurements of a linear forcesensor associated with the gripper 352, the robotic arm 304, and/or thegripper jaws 356 during an initial lift action following the grippingaction. An expected measurement profile (illustrated using dashed linesin the force magnitude v. time plot in FIG. 3B) can correspond to themeasured downward force rising to match the weight of the target object112 within a predetermined duration. However, sensor readings for animproper grip can correspond to the measured downward force failing torise to the expected levels and reaching zero or near-zero magnitude bythe end of the initial lift maneuver. In some situations, momentary lossin the grip (i.e., representative of an overall weak grip condition) cancorrespond to a negative spike or a momentary drop in the sensed linearforce.

In some embodiments, the contact sensors 226 can include torque sensorsconfigured to measure torque on the gripper jaws 356, such as when thegripper is oriented horizontally. An improper grip can cause the targetobject 112 to shift (e.g., away) from the gripper jaws 356 during alifting action, thereby changing the location of center of gravity ofthe target object 112 relative to the gripper 352. Accordingly, theamount and/or the direction of torque applied to the gripper jaws 356can change based on the shifted center of gravity. The contact measure312 can correspond to the above-described measurements according to thetype and/or location of the contact sensors 226.

System Operation

FIG. 4 is a top view illustrating an example task 402 executed by therobotic system 100 in accordance with one or more embodiments of thepresent technology. As described above, the task 402 can represent asequence of actions executed by the robotic system 100 (e.g., by one ofthe units described above, such as the transfer unit 104 of FIG. 1) toachieve a goal. As illustrated in FIG. 4, for example, the task 402 caninclude moving the target object 112 from the start location 114 (e.g.,a location on/in a receiving pallet or bin) to the task location 116(e.g., a location on/in a sorted pallet or bin).

In some embodiments, the robotic system 100 can image a predeterminedarea to identify and/or locate the start location 114. For example, therobotic system 100 can include a source scanner 412 (i.e., an instanceof the imaging devices 222 of FIG. 2) directed at a pickup area, such asan area designated for a sourcing pallet or bin and/or a region on areceiving side of the conveyor belt. The robotic system 100 can use thesource scanner 412 to generate imaging data (e.g., a captured imageand/or a point cloud) of the designated area. The robotic system 100(via, e.g., the processors 202 of FIG. 2) can implement computer visionprocesses for the imaging result to identify the different objects(e.g., boxes or cases) located in the designated area. Details of theobject identification are described below.

From the recognized objects, the robotic system 100 can select (e.g.,according to a predetermined sequence or set of rules and/or templatesof object outlines) one as the target object 112 for an execution of thetask 402. For the selected target object 112, the robotic system 100 canfurther process the imaging result to determine the start location 114and/or an initial pose. Details of the selection and the location/posedetermination are described below.

The robotic system 100 can further image and process anotherpredetermined area to identify the task location 116. In someembodiments, for example, the robotic system 100 can include anotherinstance of the imaging devices 222 (not shown) configured to generatean imaging result of a placement area, such as an area designated for asorted pallet or bin and/or a region on a sending side of the conveyorbelt. The imaging result can be processed (via, e.g., the processors202) to identify the task location 116 and/or a corresponding pose forplacing the target object 112. In some embodiments, the robotic system100 can identify (based on or without the imaging result) the tasklocation 116 according to a predetermined sequence or set of rules forstacking and/or arranging multiple objects.

Using the identified start location 114 and/or the task location 116,the robotic system 100 can operate one or more structures (e.g., therobotic arm 304 and/or the end-effector, such as the gripper 302 of FIG.3A and/or the gripper 352 of FIG. 3B) of a corresponding unit (e.g., thetransfer unit 104) to execute the task 402. Accordingly, the roboticsystem 100 (via, e.g., the processors 202) can calculate (via, e.g.,motion planning rules or algorithms) a base motion plan 422 thatcorresponds to one or more actions that will be implemented by thecorresponding unit to execute the task 402. For example, the base motionplan 422 for the transfer unit 104 can include positioning theend-effector for pickup, gripping the target object 112, lifting thetarget object 112, transferring the target object 112 from above thestart location 114 to above the task location 116, lowering the targetobject 112, and releasing the target object 112. Also, the base motionplan 422 can include only the actions necessary to successfully completethe task 402, such as for ideal conditions (e.g., without anyinterruptions, errors, unexpected external influences, etc.) orexecutions.

In some embodiments, the robotic system 100 can calculate the basemotion plan 422 by determining a sequence of commands and/or settingsfor one or more of the actuation devices 212 of FIG. 2 that operate therobotic arm 304 and/or the end-effector. For example, the robotic system100 can use the processors 202 to calculate the commands and/or settingsof the actuation devices 212 for manipulating the end-effector and therobotic arm 304 to place the gripper at a particular location about thestart location 114, engage and grab the target object 112 with theend-effector, place the end-effector at a particular location about thetask location 116, and release the target object 112 from theend-effector. The robotic system 100 can execute the actions forcompleting the task 402 by operating the actuation devices 212 accordingto the determined sequence of commands and/or settings.

In some embodiments, the task 402 can include scanning (e.g., scanning abarcode or a QR code) the target object 112, such as for product loggingpurposes and/or for further identifying the target object 112. Forexample, the robotic system 100 can include an object scanner 416 (e.g.,a further instance of the imaging devices 222, such as a barcode scanneror a QR code scanner) configured to scan the target object 112,typically at a location between the pickup area and the placement area.Accordingly, the robotic system 100 can calculate the base motion plan422 to place the target object 112 at a scanning location with apredetermined pose such that a portion or a surface of the target object112 is presented to the object scanner 416.

In executing the actions for the task 402, the robotic system 100 cantrack a current location 424 (e.g., a set of coordinates correspondingto a grid used by the robotic system 100) of the target object 112. Forexample, the robotic system 100 (via, e.g., the processors 202) cantrack the current location 424 according to data from the positionsensors 224 of FIG. 2. The robotic system 100 can locate one or moreportions of the robotic arm 304 (e.g., the structural members and/or thejoints thereof) in the kinetic chain according to the data from theposition sensors 224. The robotic system 100 can further calculate thelocation and orientation of the end-effector, and thereby the currentlocation 424 of the target object 112 held by the end-effector, based onthe location and orientation of the robotic arm 304. Also, the roboticsystem 100 can track the current location 424 based on processing othersensor readings (e.g., force readings or accelerometer readings), theexecuted actuation commands/settings and/or associated timings, or acombination thereof according to a dead-reckoning mechanism.

Also, in executing the actions for the task 402, the robotic system 100(via, e.g., the contact sensors 226) can determine the contact measure312 of FIG. 3A/FIG. 3B. The robotic system 100 can determine or samplethe contact measure 312 at various times, such as after executing aportion of the base motion plan 422 (e.g., a gripping action, adisplacing action, and/or a rotating action), according to apredetermined sampling interval or timing, or a combination thereof.

Based on the contact measure 312, the robotic system 100 can executedifferent actions to complete the task 402. In other words, the roboticsystem 100 can implement granular control/manipulation of the targetobject 112 according to the contact measure 312. For example, when orwhile the contact measure 312 satisfies a first threshold, the roboticsystem 100 can implement the base motion plan 422. When the contactmeasure 312 fails to satisfy (e.g., falls below) the first threshold,the robotic system 100 can deviate from the base motion plan 422 andexecute one or more additional and/or different actions. For example,when the contact measure 312 is below a gripping threshold afterimplementing the gripping action (e.g., by pressing the suction cups 306of FIG. 3A into the target object 112 or by applying the compressingforces via the gripper jaws 356 of FIG. 3B on opposing sides of thetarget object 112), the robotic system 100 can re-execute the grippingaction after releasing the target object 112 and/or adjusting theposition of the end-effector. The robotic system 100 can subsequentlydetermine the contact measure 312 and repeat the regripping process upto a predetermined limit if the contact measure 312 remains below thegripping threshold. If the regripping attempt results in the contactmeasure 312 that satisfies the gripping threshold, the robotic system100 can continue with the remaining portions of the base motion plan422. In some embodiments, if the robotic system 100 fails tosufficiently grip the target object 112 after a limited number ofattempts, the robotic system 100 can drop and leave the target object112 and execute the task on a different object (e.g., identifying adifferent object as the target object 112 for the next task).

Also, the robotic system 100 can deviate from the base motion plan 422when the contact measure 312 falls below a transit threshold duringmanipulation of the target object 112 (e.g., after executing thegripping action). In some embodiments, the robotic system 100 canexecute a subsequent action (e.g., a controlled drop) based on thecurrent location 424. For example, the robotic system 100 canintentionally lower and/or release the target object 112 when thecurrent location 424 of the target object 112 is above/within one ormore predetermined areas.

In some embodiments, the predetermined areas designated for thecontrolled drop action can include a source drop area 432, a destinationdrop area 434, and/or one or more transit drop areas 436. The sourcedrop area 432 can correspond to (e.g., overlap with or be offset inwardby a predetermined distance from) an area enclosed by the boundaries ofthe pickup area, such as the edges of the pallet or walls of thebin/cage. Similarly, the destination drop area 434 can correspond to theboundaries of the placement area. The transit drop areas 436 can includeareas between the pickup area and the placement area where the roboticsystem 100 can drop or place the target object 112 such that the objectwill not interfere with execution of the subsequent tasks. For theexample illustrated in FIG. 4, the transit drop areas 436 can be beforeand/or after (i.e., in moving from the pickup area to the placementarea) the object scanner 416.

Accordingly, when the contact measure 312 fails to satisfy a threshold,the robotic system 100 can calculate an adjusted drop location 442 inone of the drop areas for placing the target object 112. The roboticsystem 100 can identify the adjusted drop location 442 as a locationbetween the current location 424 and the task location 116 that hassufficient space for placing the target object 112. The robotic system100 can identify the adjusted drop location 442 similarly as the tasklocation 116. Based on the identified adjusted drop location 442 and thecurrent location 424, the robotic system can calculate an adjustedmotion plan 444 for moving the target object 112 and placing it at theadjusted drop location 442. Details regarding the identification of theadjusted drop location 442 and the calculation of the adjusted motionplan 444 are described below.

Operational Flow

FIG. 5 is a flow diagram for a method 500 of operating the roboticsystem 100 of FIG. 1 in accordance with one or more embodiments of thepresent technology. The method 500 can be for implementing granularcontrol/manipulation of the target object 112 of FIG. 1 according to thecontact measure 312 of FIG. 3A/FIG. 3B. In other words, the method 500allows the robotic system 100 to follow and/or deviate from (e.g.,perform other actions in addition to and/or instead of) the base motionplan 422 of FIG. 4 according to the contact measure 312. The method 500can be implemented based on executing the instructions stored on one ormore of the storage devices 204 of FIG. 2 with one or more of theprocessors 202 of FIG. 2.

At block 502, the robotic system 100 can scan designated areas. In someembodiments, the robotic system 100 can use (via, e.g., commands/promptssent by the processors 202) one or more of the imaging devices 222 ofFIG. 2 (e.g., the source scanner 412 of FIG. 4 and/or other areascanners) to generate imaging results (e.g., captured digital imagesand/or point clouds) of one or more designated areas, such as the pickuparea and/or the drop area (e.g., the source drop area 432 of FIG. 4, thedestination drop area 434 of FIG. 4, and/or the transit drop area 436 ofFIG. 4).

At block 504, the robotic system 100 can identify the target object 112of FIG. 1 and associated locations (e.g., the start location 114 of FIG.1 and/or the task location 116 of FIG. 1). In some embodiments, forexample, the robotic system 100 (via, e.g., the processors 202) cananalyze the imaging results according to a pattern recognition mechanismand/or a set of rules to identify object outlines (e.g., perimeter edgesor surfaces). The robotic system 100 can further identify groupings ofobject outlines (e.g., according to predetermined rules and/or posetemplates) as corresponding to each unique instance of objects. Forexample, the robotic system 100 can identify the groupings of the objectoutlines that correspond to a pattern (e.g., same values or varying at aknown rate/pattern) in color, brightness, depth/location, or acombination thereof across the object lines. Also, for example, therobotic system 100 can identify the groupings of the object outlinesaccording to predetermined shape/pose templates defined in the masterdata.

From the recognized objects in the pickup location, the robotic system100 can select (e.g., according to a predetermined sequence or set ofrules and/or templates of object outlines) one as the target object 112.For example, the robotic system 100 can select the target object 112 asthe object located on top, such as according to the point cloudrepresenting the distances/positions relative to a known location of thesource scanner 412). Also, for example, the robotic system 100 canselect the target object 112 as the object located at a corner/edge andhave two or more surfaces that are exposed/shown in the imaging results.Further, the robotic system 100 can select the target object 112according to a predetermined pattern (e.g., left to right, nearest tofurthest, etc. relative to a reference location).

For the selected target object 112, the robotic system 100 can furtherprocess the imaging result to determine the start location 114 and/or aninitial pose. For example, the robotic system 100 can determine theinitial pose of the target object 112 based on selecting from multiplepredetermined pose templates (e.g., different potential arrangements ofthe object outlines according to corresponding orientations of theobject) the one that corresponds to a lowest difference measure whencompared to the grouping of the object outlines. Also, the roboticsystem 100 can determine the start location 114 by translating alocation (e.g., a predetermined reference point for the determined pose)of the target object 112 in the imaging result to a location in the gridused by the robotic system 100. The robotic system 100 can translate thelocations according to a predetermined calibration map.

In some embodiments, the robotic system 100 can process the imagingresults of the drop areas to determine open spaces between objects. Therobotic system 100 can determine the open spaces based on mapping theobject lines according to a predetermined calibration map thattranslates image locations to real-world locations and/or coordinatesused by the system. The robotic system 100 can determine the open spacesas the space between the object lines (and thereby object surfaces)belonging to different groupings/objects. In some embodiments, therobotic system 100 can determine the open spaces suitable for the targetobject 112 based on measuring one or more dimensions of the open spacesand comparing the measured dimensions to one or more dimensions of thetarget object 112 (e.g., as stored in the master data). The roboticsystem 100 can select one of the suitable/open spaces as the tasklocation 116 according to a predetermined pattern (e.g., left to right,nearest to furthest, bottom to top, etc. relative to a referencelocation).

In some embodiments, the robotic system 100 can determine the tasklocation 116 without or in addition to processing the imaging results.For example, the robotic system 100 can place the objects at theplacement area according to a predetermined sequence of actions andlocations without imaging the area. Also, for example, the roboticsystem 100 can process the imaging result for performing multiple tasks(e.g., transferring multiple objects, such as for objects located on acommon layer/tier of a stack).

At block 506, the robotic system 100 can calculate a base plan (e.g.,the base motion plan 422 of FIG. 4) for executing the task 402 of FIG. 4for the target object 112. For example, the robotic system 100 cancalculate the base motion plan 422 based on calculating a sequence ofcommands or settings, or a combination thereof, for the actuationdevices 212 of FIG. 2 that will operate the robotic arm 304 of FIG.3A/FIG. 3B and/or the end-effector (e.g., the gripper 302 of FIG. 3Aand/or the gripper 352 of FIG. 3B). For some tasks, the robotic system100 can calculate the sequence and the setting values that willmanipulate the robotic arm 304 and/or the end-effector to transfer thetarget object 112 from the start location 114 to the task location 116.The robotic system 100 can implement a motion planning mechanism (e.g.,a process, a function, an equation, an algorithm, acomputer-generated/readable model, or a combination thereof) configuredto calculate a path in space according to one or more constraints,goals, and/or rules. For example, the robotic system 100 can use A*algorithm, D* algorithm, and/or other grid-based searches to calculatethe path through space for moving the target object 112 from the startlocation 114 to the task location 116. The motion planning mechanism canuse a further process, function, or equation, and/or a translationtable, to convert the path into the sequence of commands or settings, orcombination thereof, for the actuation devices 212. In using the motionplanning mechanism, the robotic system 100 can calculate the sequencethat will operate the robotic arm 304 and/or the end-effector and causethe target object 112 to follow the calculated path.

At block 508, the robotic system 100 can begin executing the base plan.The robotic system 100 can begin executing the base motion plan 422based on operating the actuation devices 212 according to the sequenceof commands or settings or combination thereof. The robotic system 100can execute a first set of actions in the base motion plan 422. Forexample, the robotic system 100 can operate the actuation devices 212 toplace the end-effector at a calculated location and/or orientation aboutthe start location 114 for gripping the target object 112 as illustratedin block 552. At block 554, the robotic system 100 can operate theactuation devices 212 to have the end-effector (e.g., the gripper 302and/or the gripper 352) engage and grip the target object 112. In someembodiments, as illustrated at block 556, the robotic system 100 canperform an initial lift by moving the end-effector up by a predetermineddistance. In some embodiments, the robotic system 100 can reset orinitialize an iteration counter ‘i’ used to track a number of grippingactions.

At block 510, the robotic system 100 can measure the established grip.The robotic system 100 can measure the established grip based ondetermining the contact measure 312 of FIG. 3A/FIG. 3B using one or moreof the contact sensors 226 of FIG. 2. The robotic system 100 candetermine the contact measure 312 while executing the base motion plan422, such as after gripping the target object 112 (block 554) and/orafter performing the initial lift (block 556). The robotic system 100can determine the contact measure 312 by using one or more of thecontact sensors 226 to measure a force, a torque, a pressure, or acombination thereof at one or more locations on the robotic arm 304, oneor more locations on the end-effector, or a combination thereof. In someembodiments, such as for the grip established by the gripper 302 (e.g.,a suction gripper, including the suction cups 306 of FIG. 3A), thecontact measure 312 can correspond to a quantity, a location, or acombination thereof of the suction cups 306 contacting a surface of thetarget object 112 and holding a vacuum condition therein. In someembodiments, such as for the grip established by the gripper 352 (e.g.,an impactive gripper, including the gripper jaws 356 of FIG. 3B), thecontact measure 312 can correspond to a shift in the target object 112relative to the gripper jaws 356.

At decision block 512, the robotic system 100 can compare the measuredgrip to a threshold (e.g., an initial grip threshold). For example, therobotic system 100 can compare the contact measure 312 to apredetermined threshold. In other words, the robotic system 100 candetermine whether the contact/grip is sufficient to continuemanipulating (e.g., lifting, transferring, and/or reorienting) thetarget object 112.

When the measured grip fails to satisfy the threshold, the roboticsystem 100 can evaluate whether the iteration count for regripping thetarget object 112 has reached an iteration threshold, as illustrated atdecision block 514. While the iteration count is less than the iterationthreshold, the robotic system 100 can deviate from the base motion plan422 when the contact measure 312 fails to satisfy (e.g., is below) thethreshold. Accordingly, at block 520, the robotic system 100 can operatethe robotic arm 304 and/or the end-effector to execute a regrippingaction not included in the base motion plan 422. For example, theregripping action can include a predetermined sequence of commands orsettings, or a combination thereof, for the actuation devices 212 thatwill cause the robotic arm 304 to lower the end-effector (e.g., inreversing the initial lift) and/or cause the end-effector to release thetarget object 112 and regrip the target object 112. In some embodiments,the predetermined sequence can further operate the robotic arm 304 toadjust a position of the gripper after releasing the target object andbefore regripping it. In performing the regripping action, the roboticsystem 100 can pause execution of the base motion plan 422. Afterexecuting the regripping action, the robotic system 100 can incrementthe iteration count.

After regripping the object, the robotic system 100 can measure theestablished grip as described above for block 510 and evaluate theestablished grip as described above for block 512. The robotic system100 can attempt to regrip the target object 112 as described above untilthe iteration count reaches the iteration threshold. When the iterationcount reaches the iteration threshold, the robotic system 100 can stopexecuting the base motion plan 422, as illustrated at block 516. In someembodiments, the robotic system 100 can solicit operator input, asillustrated at block 518. For example, the robotic system 100 cangenerate an operator notifier (e.g., a predetermined message) via thecommunication devices 206 of FIG. 2 and/or the input-output devices 208of FIG. 2. The robotic system 100 can process the task 402 and/or thebase motion plan 422 according to the operator input. In someembodiments, the robotic system 100 can cancel or delete the base motionplan 422, record a predetermined status (e.g., an error code) for thecorresponding task 402, or perform a combination thereof. In someembodiments, the robotic system 100 can reinitiate the process byimaging the pickup/task areas (block 502) and/or identifying anotheritem in the pickup area as the target object (block 504) as describedabove.

When the measured grip satisfies the threshold, the robotic system 100can continue executing remaining portions/actions of the base motionplan 422, as illustrated at block 522. Similarly, when the contactmeasure 312 satisfies the threshold after regripping the target object112, the robotic system 100 can resume execution of the paused basemotion plan 422. Accordingly, the robotic system 100 can continueexecuting the sequenced actions (i.e., following the grip and/or theinitial lift) in the base motion plan 422 by operating the actuationdevices 212 and/or the transport motor 214 of FIG. 2 according to theremaining sequence of commands and/or settings. For example, the roboticsystem 100 can transfer (e.g., vertically and/or horizontally) and/orreorient the target object 112 according to the base motion plan 422.

While executing the base motion plan 422, the robotic system 100 cantrack the current location 424 and/or the current orientation of thetarget object 112. The robotic system 100 can track the current location424 according to outputs from the position sensors 224 of FIG. 2 tolocate one or more portions of the robotic arm 304 and/or theend-effector. In some embodiments, the robotic system 100 can track thecurrent location 424 by processing the outputs of the position sensors224 with a computer-generated model, a process, an equation, a positionmap, or a combination thereof. Accordingly, the robotic system 100 cancombine the positions or orientations of the joints and the structuralmembers and further map the positions to the grid to calculate and trackthe current location 424. In some embodiments, the base motion plan 422can use a multilaterating system. For example, the robotic system 100can include multiple beacon sources. The robotic system 100 can measurethe beacon signals at one or more locations in the robotic arm 304and/or the end-effector and calculate separation distances between thesignal sources and the measured location using the measurements (e.g.,signal strength, time stamp or propagation delay, and/or phase shift).The robotic system 100 can map the separation distances to knownlocations of the signal sources and calculate the current location ofthe signal-receiving location as the location where the mappedseparation distances overlap.

At decision block 524, the robotic system 100 can determine whether thebase plan has been fully executed to the end. For example, the roboticsystem 100 can determine whether all of the actions (e.g., the commandsand/or the settings) in the base motion plan 422 have been completed.Also, the robotic system 100 can determine that the base motion plan 422is finished when the current location 424 matches the task location 116.When the robotic system 100 has finished executing the base plan, therobotic system 100 can reinitiate the process by imaging the pickup/taskareas (block 502) and/or identifying another item in the pickup area asthe target object (block 504) as described above.

Otherwise, at block 526, the robotic system 100 can measure the grip(i.e., by determining the contact measure 312) during transfer of thetarget object 112. In other words, the robotic system 100 can determinethe contact measure 312 while executing the base motion plan 422. Insome embodiments, the robotic system 100 can determine the contactmeasure 312 according to a sampling frequency or at predetermined times.In some embodiments, the robotic system 100 can determine the contactmeasure 312 before and/or after executing a predetermined number ofcommands or settings with the actuation devices 212. For example, therobotic system 100 can sample the contact sensors 226 after or during aspecific category of maneuvers, such as for lifts or rotations. Also,for example, the robotic system 100 can sample the contact sensors 226when a direction and/or a magnitude of an accelerometer output matchesor exceeds a predetermined threshold that represents a sudden or fastmovement. The robotic system 100 can determine the contact measure 312using one or more processes described above (e.g., for block 510).

In some embodiments, the robotic system 100 can determine theorientation of the gripper and/or the target object 112 and adjust thecontact measure accordingly. The robotic system 100 can adjust thecontact measure based on the orientation to account for a directionalrelationship between a sensing direction for the contact sensor andgravitational force applied to the target object according to theorientation. For example, the robotic system 100 can calculate an anglebetween the sensing direction and a reference direction (e.g., “down” orthe direction of the gravitational force) according to the orientation.The robotic system 100 can scale or multiply the contact measureaccording to a factor and/or a sign that corresponds to the calculatedangle.

At decision block 528, the robotic system 100 can compare the measuredgrip to a threshold (e.g., a transfer grip threshold). In someembodiments, the transfer grip threshold can be less than or equal tothe initial grip threshold associated with evaluating an initial (e.g.,before transferring) grip on the target object 112. Accordingly, therobotic system 100 can enforce a stricter rule for evaluating the gripbefore initiating transfer of the target object 112. The thresholdrequirement for the grip can be higher initially since contactsufficient for picking up the target object 112 is likely to besufficient for transferring the target object 112.

When the measured grip satisfies (e.g., is not less than) the threshold,the robotic system 100 can continue executing the base plan asillustrated at block 522 and described above. When the measured gripfails to satisfy (e.g., is less than) the threshold, the robotic system100 can deviate from the base motion plan 422 and execute one or moreresponsive actions as illustrated at block 530. Accordingly, when themeasured grip is insufficient in light of the threshold, the roboticsystem 100 can operate the robotic arm 304, the end-effector, or acombination thereof according to commands and/or settings not includedin the base motion plan 422. In some embodiments, the robotic system 100can execute different commands and/or settings based on the currentlocation 424.

For illustrative purposes, the response actions will be described usinga controlled drop. However, it is understood that the robotic system 100can execute other actions, such as by stopping execution of the basemotion plan 422 as illustrated at block 516 and/or by solicitingoperator input as illustrated at block 518.

The controlled drop includes one or more actions for placing the targetobject 112 in one of the drop areas (e.g., instead of the task location116) in a controlled manner (i.e., based on lowering and/or releasingthe target object 112 and not as a result of a complete grip failure).In executing the controlled drop, the robotic system 100 can dynamically(i.e., in real time and/or while executing the base motion plan 422)calculate different locations, maneuvers or paths, and/or actuationdevice commands or settings according to the current location 424.

At block 562, the robotic system 100 can calculate the adjusted droplocation 442 of FIG. 4 and/or an associated pose for placing the targetobject 112. In calculating the adjusted drop location 442, the roboticsystem 100 can identify the drop area (e.g., the source drop area 432 ofFIG. 4, the destination drop area 434 of FIG. 4, or the transit droparea 436 of FIG. 4) nearest to and/or ahead (e.g., between the currentlocation 424 and the task location 116) of the current location 424. Therobotic system 100 can identify the suitable drop area based oncomparing the current location 424 to boundaries that define the dropareas. In some embodiments, when the current location 424 is within oneof the drop areas (e.g., such as when the target object 112 is stillabove the source pallet/bin or the target pallet/bin), the roboticsystem 100 can calculate the adjusted drop location 442 as the currentlocation 424. In some embodiments, when the current location 424 iswithin one of the drop areas, the robotic system 100 can calculate theadjusted drop location 442 based on adding a predetermined offsetdistance and/or direction to the current location 424, such as forplacing the target object 112 away from a commonly used corridor.

Also, when the current location 424 is between (i.e., not within) thedrop areas, the robotic system 100 can calculate distances to the dropareas (e.g., distances to representative reference locations for thedrop areas). Accordingly, the robotic system 100 can identify the droparea that is nearest to the current location 424 and/or ahead of thecurrent location 424. Based on the identified drop area, the roboticsystem 100 can calculate a location therein as the adjusted droplocation 442. In some embodiments, the robotic system 100 can calculatethe adjusted drop location 442 based on selecting a location accordingto a predetermined order (e.g., left to right, bottom to top, and/orfront to back relative to a reference location).

In some embodiments, the robotic system 100 can calculate distances fromthe current location 424 to open spaces (e.g., as identified in block504 and/or tracked according to ongoing placements of objects) withinthe drop areas. The robotic system 100 can select the open space that isahead of the current location 424 and/or nearest to the current location424 as the adjusted drop location 442.

In some embodiments, prior to selecting the drop area and/or the openspace, the robotic system 100 can use a predetermined process and/orequation to translate the contact measure 312 to a maximum transferdistance. For example, the predetermined process and/or equation canestimate based on various values of the contact measure 312 acorresponding maximum transfer distance and/or a duration before acomplete grip failure. Accordingly, the robotic system 100 can filterout the available drop areas and/or the open spaces that are fartherthan the maximum transfer distance from the current location 424. Insome embodiments, when the robotic system 100 fails to identifyavailable drop areas and/or open spaces (e.g., when the accessible dropareas are full), the robotic system 100 can stop executing the basemotion plan 422, as illustrated at block 516, and/or solicit operatorinput, as illustrated at block 518.

At block 566, the robotic system 100 can calculate the adjusted motionplan 444 for transferring the target object 112 from the currentlocation 424 to the adjusted drop location 442. The robotic system 100can calculate the adjusted motion plan 444 in a way similar to thatdescribed above for block 506. For example, the robotic system 100 canuse A* or D* to calculate a path from the current location 424 to theadjusted drop location 442 and convert the path into a sequence ofcommands or settings, or a combination thereof, for the actuationdevices 212 that will operate the robotic arm 304 and/or theend-effector to maneuver the target object 112 to follow the path.

At block 568, the robotic system 100 can execute the adjusted motionplan 444 in addition to and/or instead of the base motion plan 422. Forexample, the robotic system 100 can operate the actuation devices 212according to the sequence of commands or settings or combinationthereof, thereby maneuvering the robotic arm 304 and/or the end-effectorto cause the target object 112 to move according to the path.

In some embodiments, the robotic system 100 can pause execution of thebase motion plan 422 and execute the adjusted motion plan 444. Once thetarget object 112 is placed at the adjusted drop location 442 based onexecuting the adjusted motion plan 444 (i.e., completing execution ofthe controlled drop), in some embodiments, the robotic system 100 canattempt to regrip the target object 112 as described above for block 520and then measure the established grip as described above for block 510.In some embodiments, the robotic system 100 can attempt to regrip thetarget object 112 up to an iteration limit as described above. If thecontact measure 312 satisfies the initial grip threshold, the roboticsystem 100 can reverse the adjusted motion plan 444 (e.g., return to thepaused point/location) and continue executing the remaining portions ofthe paused base motion plan 422. In some embodiments, the robotic system100 can update and recalculate the adjusted motion plan 444 from thecurrent location 424 (after regripping) to the task location 116 andexecute the adjusted motion plan 444 to finish executing the task 402.

In some embodiments, at block 570 the robotic system 100 can update anarea log (e.g., a record of open spaces and/or placed objects) for theaccessed drop area to reflect the placed target object 112. For example,the robotic system 100 can regenerate the imaging results for thecorresponding drop area. In some embodiments, the robotic system 100 cancancel the remaining actions of the base motion plan 422 after executingthe controlled drop and placing the target object 112 at the adjusteddrop location 442. In one or more embodiments, the transit drop area 436can include a pallet or a bin placed on top of one of the transportunits 106 of FIG. 1. At a designated time (e.g., when the pallet/bin isfull and/or when the incoming pallet/bin is delayed), the correspondingtransport unit can go from the drop area to the pickup area.Accordingly, the robotic system 100 can reimplement the method 500,thereby reidentifying the dropped items as the target object 112 andtransferring them to the corresponding task location 116.

Once the target object 112 has been placed at the adjusted drop location442, the robotic system 100 can repeat the method 500 for a new targetobject. For example, the robotic system 100 can determine the nextobject in the pickup area as the target object 112, calculate a new basemotion plan to transfer the new target object, etc.

In some embodiments, the robotic system 100 can include a feedbackmechanism that updates the path calculating mechanism based on thecontact measure 312. For example, as the robotic system 100 implementsthe actions to regrip the target object 112 with adjusted positions(e.g., as described above for block 520), the robotic system 100 canstore the position of the end-effector that produced the contact measure312 that satisfied the threshold (e.g., as described above for block512). The robotic system 100 can store the position in association withthe target object 112. The robotic system 100 can analyze the storedpositions (e.g., using a running window for analyzing a recent set ofactions) for gripping the target object 112 when the number of gripfailures and/or successful regrip actions reach a threshold. When apredetermined number of regrip actions occur for a specific object, therobotic system 100 can update the motion planning mechanism to place thegripper at a new position (e.g., position corresponding to the highestnumber of successes) relative to the target object 112.

Based on the operations represented in block 510 and/or block 526 therobotic system 100 (via, e.g., the processors 202) can track a progressof executing the base motion plan 422. In some embodiments, the roboticsystem 100 can track the progress according to horizontal transfer ofthe target object 112. For example, as illustrated in FIG. 5, therobotic system 100 can track the progress based on measuring theestablished grip (block 510) before initiating the horizontal transferand based on measuring the grip during transfer (block 526) afterinitiating the horizontal transfer. Accordingly, the robotic system 100can selectively generate a new set (i.e., different from the base motionplan 422) of actuator commands, actuator settings, or a combinationthereof based on the progress as described above.

In other embodiments, for example, the robotic system 100 can track theprogress based on tracking the commands, the settings, or a combinationthereof that has been communicated to and/or implemented by theactuation devices 212. Based on the progress, the robotic system 100 canselectively generate the new set of actuator commands, actuatorsettings, or a combination thereof to execute the regrip response actionand/or the controlled drop response action. For example, when theprogress is before any horizontal transfer of the target object 112, therobotic system 100 can select the initial grip threshold and execute theoperations represented in blocks 512 (via, e.g., function calls or jumpinstructions) and onward. Also, when the progress is after thehorizontal transfer of the target object 112, the robotic system 100 canselect the transfer grip threshold and execute the operationsrepresented in blocks 528 (via, e.g., function calls or jumpinstructions) and onward.

Implementing granular control/manipulation of the target object 112(i.e., choosing to implement the base motion plan 422 or deviate fromit) according to the contact measure 312 provides improved efficiency,speed, and accuracy for transferring the objects. For example,regripping the target object 112 when the contact measure 312 is belowthe initial grip threshold decreases the likelihood of grip failureoccurring during transfer, which decreases the number of objects lost orunintentionally dropped during transfer. Moreover, each lost objectrequires human interaction to correct the outcome (e.g., move the lostobject out of the motion path for subsequent tasks, inspect the lostobject for damages, and/or complete the task for the lost object). Thus,reducing the number of lost objects reduces the human effort necessaryto implement the tasks and/or the overall operation.

Moreover, placing the target object 112 in designated areas when thecontact measure 312 is below the transfer grip threshold reduces thenumber of untracked obstacles and damaged items. Based on calculatingthe adjusted drop location 442 and executing the controlled drop, thetarget object 112 can be placed at known locations. Accordingly, thenumber of lost objects that end up in random untracked locations isreduced, which further reduces the likelihood of a lost object ending upat a location that blocks or hinders execution of subsequent tasks.Moreover, the robotic system 100 can avoid frequently used path segmentsin calculating the adjusted drop location 442 as described above,thereby further reducing the impact of insufficient grips. Additionally,since the target object 112 is placed in a controlled manner instead ofbeing dropped from a height with momentum, the target object 112contacts the placement location with less force. As such, executing thecontrolled drop greatly reduces the damages caused by losing theobjects.

CONCLUSION

The above Detailed Description of examples of the disclosed technologyis not intended to be exhaustive or to limit the disclosed technology tothe precise form disclosed above. While specific examples for thedisclosed technology are described above for illustrative purposes,various equivalent modifications are possible within the scope of thedisclosed technology, as those skilled in the relevant art willrecognize. For example, while processes or blocks are presented in agiven order, alternative implementations may perform routines havingsteps, or employ systems having blocks, in a different order, and someprocesses or blocks may be deleted, moved, added, subdivided, combined,and/or modified to provide alternative or sub-combinations. Each ofthese processes or blocks may be implemented in a variety of differentways. Also, while processes or blocks are at times shown as beingperformed in series, these processes or blocks may instead be performedor implemented in parallel, or may be performed at different times.Further, any specific numbers noted herein are only examples;alternative implementations may employ differing values or ranges.

These and other changes can be made to the disclosed technology in lightof the above Detailed Description. While the Detailed Descriptiondescribes certain examples of the disclosed technology as well as thebest mode contemplated, the disclosed technology can be practiced inmany ways, no matter how detailed the above description appears in text.Details of the system may vary considerably in its specificimplementation, while still being encompassed by the technologydisclosed herein. As noted above, particular terminology used whendescribing certain features or aspects of the disclosed technologyshould not be taken to imply that the terminology is being redefinedherein to be restricted to any specific characteristics, features, oraspects of the disclosed technology with which that terminology isassociated. Accordingly, the invention is not limited, except as by theappended claims. In general, the terms used in the following claimsshould not be construed to limit the disclosed technology to thespecific examples disclosed in the specification, unless the aboveDetailed Description section explicitly defines such terms.

Although certain aspects of the invention are presented below in certainclaim forms, the applicant contemplates the various aspects of theinvention in any number of claim forms. Accordingly, the applicantreserves the right to pursue additional claims after filing thisapplication to pursue such additional claim forms, in either thisapplication or in a continuing application.

We claim:
 1. A method for operating a robotic system, the methodcomprising: obtaining a base motion plan that includes a first set ofcommands and/or settings configured to operate a robotic arm and/or agripper to transfer a target object from a start location to a tasklocation; tracking a progress during execution of the base motion plan,wherein tracking the progress includes tracking a current location ofthe target object during execution of the base motion plan; receiving acontact measure during the execution of the base motion plan, whereinthe contact measure represents a force, a torque, and/or a pressuremeasured by a contact sensor; determining whether the contact measurefails to satisfy a threshold; and selectively generating a second set ofcommands and/or settings based on the determination and the trackedprogress, wherein: the second set deviates from the base motion plan andexecutes one or more response actions that are different than the basemotion plan, selectively generating the second set includes calculatingan adjusted motion plan that deviates from the base motion plan, andcalculating the adjusted motion plan includes selecting a drop areabased on comparing the current location to a source drop area, adestination drop area, and/or a transit drop area and calculating theadjusted motion plan for transferring the target object from the currentlocation to an adjusted drop location instead of the task location. 2.The method of claim 1, wherein: the contact measure represents theamount of grip on the target object provided by a set of suction cups ofa suction gripper; and determining whether the contact measure fails tosatisfy the threshold includes comparing the contact measure to thethreshold that represents a quantity, a location, or a combinationthereof of the suction cups contacting a surface of the target objectand holding a vacuum condition therein.
 3. The method of claim 1,wherein: the contact measure represents the amount of grip on the targetobject provided by an impactive gripper having a set of jaws applyingcompressing forces on opposing portions of the target object; anddetermining whether the contact measure fails to satisfy the thresholdincludes comparing the contact measure to the threshold configured todetect a shift in the target object relative to the jaws.
 4. The methodof claim 1, wherein selecting the drop area includes: determiningwhether the current location is within the source drop area that isassociated with boundaries of a container or a pallet at the startlocation; and selecting the source drop area based on whether thecurrent location is within the source drop area for placing the targetobject at the adjusted drop location that is different from the startlocation.
 5. The method of claim 1, wherein selecting the drop areaincludes: determining whether the current location is within thedestination drop area that is associated with boundaries of a containeror a pallet at the task location; and selecting the destination droparea based on the determination for intentionally placing the targetobject at the adjusted drop location therein.
 6. The method of claim 1,wherein selecting the drop area includes: determining whether thecurrent location is within the transit drop area that is between thesource drop area and the destination drop area; and selecting thetransit drop area based on the determination for intentionally placingthe target object at the adjusted drop location therein.
 7. The methodof claim 1, wherein: the contact measure represents a measurement of theamount of grip after initiating horizontal transfer of the targetobject; and determining whether the contact measure fails to satisfy thethreshold includes comparing the contact measure to a transfer thresholdthat is less than an initial grip threshold associated with initiallygripping the target object before horizontally transferring the targetobject.
 8. The method of claim 1, wherein: the contact measurerepresents the amount of grip measured after gripping the target objectand before initiating horizontal transfer of the target object; the gripthreshold is an initial threshold; and the second set includes commandsand/or settings that deviate from the base motion plan to regrip thetarget object.
 9. The method of claim 8, further comprising: executing afirst portion of the base motion plan for placing the gripper about thestart location and for initially gripping the target object with thegripper; wherein: receiving the contact measure includes receiving thecontact measure determined after initially gripping the target object;and the commands and/or the settings of the second set are configured tooperate the robotic arm and/or the gripper to: release the target objectfrom the gripper, and regrip the target object with the gripper.
 10. Themethod of claim 9, wherein the second set further includes commandsand/or settings for adjusting a position of the gripper between therelease and the regrip.
 11. The method of claim 9, further comprising:pausing execution of the base motion plan when the contact measure failsto satisfy the initial threshold; and resuming execution of the basemotion plan after regripping the target object.
 12. A tangible,non-transient computer-readable medium having processor instructionsstored thereon that, when executed by a robotic system via one or moreprocessors thereof, cause the robotic system to: obtain a base motionplan, wherein the base motion plan includes a first set of commandsand/or settings configured to operate a robotic arm and/or a gripper totransfer a target object from a start location to a task location; tracka progress during execution of the base motion plan, wherein theprogress is based on a current location of the target object; receive acontact measure during execution of the base motion plan, wherein thecontact measure represents an amount of grip of the gripper on thetarget object measured by a contact sensor and corresponds to the amountof grip after initiating horizontal transfer of the target object;select a drop area based on the current location, wherein the drop areais selected from a source drop area, a destination drop area, and/or atransit drop area; determine whether the contact measure fails tosatisfy a threshold; calculate an adjusted drop location in the selecteddrop area, wherein the adjusted drop location is calculated after thecontact measure is determined to be below the threshold; selectivelygenerate a second set of commands and/or settings based on the contactmeasure for operating the robotic arm and/or the gripper to execute oneor more response actions that deviate from the base motion plan; andcalculate an adjusted motion plan based on selectively generating thesecond set of the commands and/or the settings for transferring thetarget object from the current location to the adjusted drop location,wherein the adjusted drop location is different from the task location.13. The tangible, non-transient computer-readable medium of claim 12further comprising instructions to: determine whether the currentlocation is within the source drop area that is associated withboundaries of a container or a pallet at the start location; and selectthe source drop area based on whether the current location is within thesource drop area for placing the target object at the adjusted droplocation that is different from the start location.
 14. The tangible,non-transient computer-readable medium of claim 12 further comprisinginstructions to: determine whether the current location is within thedestination drop area that is associated with boundaries of a containeror a pallet at the task location; and select the destination drop areabased on the determination for intentionally placing the target objectat the adjusted drop location therein.
 15. The tangible, non-transientcomputer-readable medium of claim 12 further comprising instructions to:determine whether the current location is within the transit drop areathat is between the source drop area and the destination drop area; andselect the transit drop area based on the determination forintentionally placing the target object at the adjusted drop locationtherein.
 16. A robotic system, comprising: at least one processor; andat least one memory device connected to the at least one processor andhaving stored thereon instructions executable by the processor to:obtain a base motion plan, wherein the base motion plan includes a firstset of commands and/or settings, that is configured to operate a roboticarm and/or a gripper to transfer the target object from a start locationto a task location; track a progress during execution of the base motionplan, wherein the progress is based on a current location of the targetobject; receive a contact measure during the execution of the basemotion plan, wherein the contact measure represents an amount of grip ofthe gripper on the target object measured by a contact sensor andcorresponds to the amount of grip after initiating horizontal transferof the target object; select a drop area based on the current location,wherein the drop area is selected from a source drop area, a destinationdrop area, and/or a transit drop area; determine whether the contactmeasure fails to satisfy a threshold; calculate an adjusted droplocation in the selected drop area, wherein the adjusted drop locationis calculated after the contact measure is determined to be below thethreshold; selectively generate a second set of commands and/or settingsbased on the determination and the tracked progress for operating therobotic arm and/or the gripper to deviate from the base motion plan andexecute one or more response actions separate from the base motion plan;and calculate an adjusted motion plan based on selectively generatingthe second set of the commands and/or the settings for transferring thetarget object from the current location to the adjusted drop location,wherein the adjusted drop location is different from the task location.17. The robotic system of claim 16, wherein the memory device includesinstructions executable by the processor to: select the drop area as thesource drop area, the destination drop area, or the transit drop areathat overlaps with the current location intentionally placing the targetobject therein instead of the task location and instead of the startlocation, wherein: the source drop area is associated with boundaries ofa container or a pallet including the start location, the destinationdrop area is associated with boundaries of a container or a palletincluding the task location, and the transit drop area is between thesource drop area and the destination drop area.
 18. The robotic systemof claim 16, wherein the contact measure represents a force, a torque, apressure, or a combination thereof associated with a set of suction cupsor a set of jaws of the gripper used to contact and grip the targetobject.
 19. The robotic system of claim 16, wherein the memory deviceincludes instructions executable by the processor to: receive an initialcontact measure that represents the amount of grip after gripping thetarget object and before initiating horizontal transfer of the targetobject; and wherein: the second set of commands and/or settings furtherincludes regrip commands and/or regrip settings that deviate from thebase motion plan to release and regrip the target object.
 20. Therobotic system of claim 19, wherein the memory device includesinstructions executable by the processor to: execute a first set ofactions in the base motion plan to place the gripper about the startlocation and initially grip the target object with the gripper; wherein:the initial contact measure represents the amount of grip determinedafter initially gripping the target object; and the second set ofcommands and/or settings are configured to operate the robotic armand/or the gripper to: release the target object from the gripper afterexecuting the first set of actions, and regrip the target object.